Rotating Camera and Microphone Configurations

ABSTRACT

An apparatus including a first part, the first part having at least one camera configured to capture images; a second part having at least one microphone configured to capture at least one audio signal, wherein one of the first part or second part is able to move relative to the other part and the apparatus including circuitry configured to: determine a parameter associated with the move; generate at least one output audio signal based on the parameter associated with the move and the at least one audio signal.

FIELD

The present application relates to apparatus and methods for rotatingcamera and microphone configurations, but not exclusively for rotatingcamera and microphone configurations within spatial audio captureapparatus.

BACKGROUND

Spatial audio capture is a rapidly developing field of investigation.Conventionally a capture device has a microphone configuration which isfixed relative to the camera. In such configurations the spatialrelationship between the camera or cameras and the microphones is fixedand aligning the spatial audio signal and video images is a simpleoperation.

For example spatial audio which has the ability to determine audiodirections in a plane can be captured using a device comprising 3microphones and to determine audio direction in all directions can becaptured using a device comprising 4 microphones.

Audio directions can be typically analysed based on level and phase/timedifferences of microphone signals. The physical configuration affectsaudio signals coming from different directions differently and differentmicrophone locations cause sound from different directions to arrive atdifferent time to the microphones. The different arrival times TDOA(Time Difference of Arrival) can be used to determine directions usingknown methods. With the fixed distances and locations of the microphonesrelative to the camera these directions can be aligned with the cameradirection in a simple manner.

There may be in some situations a capture device which has the abilityto move or rotate the camera relative to the microphones. In suchcapture devices there is a need to be able to more efficiently handlethe audio signals generated, for example to maintain a ‘correct’alignment otherwise the difference between objects in the video imagesand the audio directions may be distracting for the user of the playbackapparatus.

SUMMARY

There is provided according to a first aspect an apparatus comprising: afirst part, the first part having at least one camera configured tocapture images; a second part having at least one microphone configuredto capture at least one audio signal, wherein one of the first part orsecond part is able to move relative to the other part and the apparatuscomprising means configured to: determine a parameter associated withthe move; generate at least one output audio signal based on theparameter associated with the move and the at least one audio signal.

The first part or the second part may be able to move relative to commonreference point.

The move may be at least one of: a rotation about an axis in commonbetween the first part and the second part; a pitch and/or yaw and/orroll between the first part and the second part; a movement of the firstpart relative to the second part; and a movement of the second partrelative to the first part.

The means may be further configured to: multiplex the at least oneoutput audio signal and the images captured by the camera; and outputthe multiplexed at least one output audio signal and the images capturedby the camera.

The first part may further have at least one further microphoneconfigured to capture at least one further audio signal, wherein themeans configured to generate at least one output audio signal based onthe parameter associated with the move and the at least one audio signalmay be configured to generate the at least one output audio signalfurther based on the at least one further audio signal.

The means configured to generate the at least one output audio signalfurther based on the at least one further audio signal may be configuredto align the at least one output audio signal and the at least onefurther audio signal based on the parameter associated with the move.

The at least one microphone may comprise at least three microphonesarranged with respect to the second part, and the means configured togenerate at least one output audio signal based on the parameterassociated with the move and the at least one audio signal may beconfigured to: obtain a parameter defining the arrangement of the atleast three microphones; obtain a parameter defining an orientation ofthe apparatus; and generate the at least one output audio signal furtherbased on the parameter defining the arrangement of the at least threemicrophones and the parameter defining an orientation of the apparatus.

The means configured to generate the at least one output audio signalfurther based on the parameter defining the arrangement of the at leastthree microphones and the parameter defining an orientation of theapparatus may be configured to generate the at least one output audiosignal for at least one frequency band based on the parameter definingthe arrangement of the at least three microphones and the parameterdefining an orientation of the apparatus.

The means configured to generate at least one output audio signal basedon the parameter associated with the move and the at least one audiosignal may be configured to align the at least one output audio signalbased on the parameter associated with the move such that the at leastone output audio signal is aligned with the camera.

The at least one microphone may be configured to receive acoustic wavesexternal of the apparatus via at least one sound port, wherein the atleast sound port comprises at least one dimension which may be modifiedbased on at least the first part relative to the second part.

The first part relative to the second part may be an angle of the firstpart relative to the second part.

The effective location of the at least one microphone may be defined bythe at least one dimension which may be modified based on at least theangle of the first part relative to the second part.

The at least one output audio signals may comprises at least one of: atleast one spatial audio signal; at least one non-spatial audio signal; amono audio signal; a beamformed audio signal; and a shotgun audiosignal.

The means configured to generate the at least one output audio signalbased on the parameter associated with the move and the at least oneaudio signal may be further configured to analyse the at least one audiosignal based on the parameter and a frequency band associated with theat least one audio signal.

The means configured to generate the at least one output audio signalbased on the parameter associated with the move and the at least oneaudio signal may be further configured to analyse the at least onefurther audio signal based on the parameter and a frequency bandassociated with the at least one audio signal.

The parameter may comprise a rotation of the first part relative to thesecond part.

The means configured to generate the at least one output audio signalbased on the parameter associated with the move and the at least oneaudio signal may be further configured to: generate a mono audio signalbased on a rotation of the first part relative to the second part and/oran orientation of the apparatus being a first configuration; andgenerate a spatial audio signal based on a rotation of the first partrelative to the second part and/or an orientation of the apparatus beinga second configuration.

The means configured to generate the at least one output audio signalbased on the parameter associated with the move, the at least one audiosignal, and the at least one further audio signal may be furtherconfigured to: generate a mono audio signal based on the at least onefurther audio signal; generate a spatial audio signal based on the atleast one audio signal; combine the mono audio signal based on the atleast one further audio signal and the spatial audio signal based on theat least one audio signal based on the parameter.

The one of the first part or second part being able to move relative tothe other part is configured to reveal the at least one microphone suchthat the at least one audio signal captured by the at least onemicrophone signal is a spatial audio signal.

According to a second aspect there is provided a method comprising:providing an apparatus, the apparatus comprising a first part, the firstpart having at least one camera configured to capture images; a secondpart having at least one microphone configured to capture at least oneaudio signal, wherein one of the first part or second part is able tomove relative to the other part;

determining a parameter associated with the move;

generating at least one output audio signal based on the parameterassociated with the move and the at least one audio signal.

The first part or the second part may be able to move relative to commonreference point.

The move may be at least one of: a rotation about an axis in commonbetween the first part and the second part; a pitch and/or yaw and/orroll between the first part and the second part; a movement of the firstpart relative to the second part; and a movement of the second partrelative to the first part.

The means may be further configured to: multiplex the at least oneoutput audio signal and the images captured by the camera; and outputthe multiplexed at least one output audio signal and the images capturedby the camera.

The first part may further have at least one further microphoneconfigured to capture at least one further audio signal, whereingenerating at least one output audio signal based on the parameterassociated with the move and the at least one audio signal may comprisegenerating the at least one output audio signal further based on the atleast one further audio signal.

Generating the at least one output audio signal further based on the atleast one further audio signal may comprise aligning the at least oneoutput audio signal and the at least one further audio signal based onthe parameter associated with the move.

The at least one microphone comprises at least three microphonesarranged with respect to the second part, and generating at least oneoutput audio signal based on the parameter associated with the move andthe at least one audio signal may comprise: obtaining a parameterdefining the arrangement of the at least three microphones; obtaining aparameter defining an orientation of the apparatus; and generating theat least one output audio signal further based on the parameter definingthe arrangement of the at least three microphones and the parameterdefining an orientation of the apparatus.

Generating the at least one output audio signal further based on theparameter defining the arrangement of the at least three microphones andthe parameter defining an orientation of the apparatus may comprisegenerating the at least one output audio signal for at least onefrequency band based on the parameter defining the arrangement of the atleast three microphones and the parameter defining an orientation of theapparatus.

Generating at least one output audio signal based on the parameterassociated with the move and the at least one audio signal may comprisealigning the at least one output audio signal based on the parameterassociated with the move such that the at least one output audio signalis aligned with the camera.

The method may comprise: receiving, by the at least one microphone,acoustic waves external of the apparatus via at least one sound port.

The method may comprise modifying at least one dimension of the at leastsound port based on at least the first part relative to the second part.

The first part relative to the second part may be an angle of the firstpart relative to the second part.

Modifying at least one dimension of the at least sound port based on atleast the first part relative to the second part may modify an effectivelocation of the at least one microphone.

The at least one output audio signal may comprise at least one of: atleast one spatial audio signal; at least one non-spatial audio signal; amono audio signal; a beamformed audio signal; and a shotgun audiosignal.

Generating the at least one output audio signal based on the parameterassociated with the move and the at least one audio signal may furthercomprise analysing the at least one audio signal based on the parameterand a frequency band associated with the at least one audio signal.

Generating the at least one output audio signal based on the parameterassociated with the move and the at least one audio signal may furthercomprise analysing the at least one further audio signal based on theparameter and a frequency band associated with the at least one audiosignal.

The parameter may comprise a rotation of the first part relative to thesecond part.

Generating the at least one output audio signal based on the parameterassociated with the move and the at least one audio signal may furthercomprise: generating a mono audio signal based on a rotation of thefirst part relative to the second part and/or an orientation of theapparatus being a first configuration; and generating a spatial audiosignal based on a rotation of the first part relative to the second partand/or an orientation of the apparatus being a second configuration.

Generating the at least one output audio signal based on the parameterassociated with the move, the at least one audio signal, and the atleast one further audio signal may further comprise: generating a monoaudio signal based on the at least one further audio signal; generatinga spatial audio signal based on the at least one audio signal; andcombining the mono audio signal based on the at least one further audiosignal and the spatial audio signal based on the at least one audiosignal based on the parameter.

The one of the first part or second part being able to move relative tothe other part is configured to reveal the at least one microphone suchthat the at least one audio signal captured by the at least onemicrophone signal is a spatial audio signal.

According to a third aspect there is provided an apparatus comprising: afirst part, the first part having at least one camera configured tocapture images; a second part having at least one microphone configuredto capture at least one audio signal, wherein one of the first part orsecond part is able to move relative to the other part; at least oneprocessor and at least one memory including a computer program code, theat least one memory and the computer program code configured to, withthe at least one processor, cause the apparatus at least to: determine aparameter associated with the move; generate at least one output audiosignal based on the parameter associated with the move and the at leastone audio signal.

The first part or the second part may be able to move relative to commonreference point.

The move may be at least one of: a rotation about an axis in commonbetween the first part and the second part; a pitch and/or yaw and/orroll between the first part and the second part; a movement of the firstpart relative to the second part; and a movement of the second partrelative to the first part.

The apparatus may be further caused to: multiplex the at least oneoutput audio signal and the images captured by the camera; and outputthe multiplexed at least one output audio signal and the images capturedby the camera.

The first part may further have at least one further microphoneconfigured to capture at least one further audio signal, wherein theapparatus caused to generate at least one output audio signal based onthe parameter associated with the move and the at least one audio signalmay be further caused to generate the at least one output audio signalfurther based on the at least one further audio signal.

The apparatus caused to generate the at least one output audio signalfurther based on the at least one further audio signal may be caused toalign the at least one output audio signal and the at least one furtheraudio signal based on the parameter associated with the move.

The at least one microphone may comprise at least three microphonesarranged with respect to the second part, and the apparatus caused togenerate at least one output audio signal based on the parameterassociated with the move and the at least one audio signal may be causedto: obtain a parameter defining the arrangement of the at least threemicrophones; obtain a parameter defining an orientation of theapparatus; and generate the at least one output audio signal furtherbased on the parameter defining the arrangement of the at least threemicrophones and the parameter defining an orientation of the apparatus.

The apparatus caused to generate the at least one output audio signalfurther based on the parameter defining the arrangement of the at leastthree microphones and the parameter defining an orientation of theapparatus may be caused to generate the at least one output audio signalfor at least one frequency band based on the parameter defining thearrangement of the at least three microphones and the parameter definingan orientation of the apparatus.

The apparatus caused to generate at least one output audio signal basedon the parameter associated with the move and the at least one audiosignal may be caused to align the at least one output audio signal basedon the parameter associated with the move such that the at least oneoutput audio signal is aligned with the camera.

The apparatus may be caused to: receive, by the at least one microphone,acoustic waves external of the apparatus via at least one sound port.

The apparatus may be caused to modify at least one dimension of the atleast sound port based on at least the first part relative to the secondpart.

The first part relative to the second part may be an angle of the firstpart relative to the second part.

The apparatus caused to modify at least one dimension of the at leastsound port based on at least the first part relative to the second partmay be caused to modify an effective location of the at least onemicrophone.

The at least one output audio signal may comprise at least one of: atleast one spatial audio signal; at least one non-spatial audio signal; amono audio signal; a beamformed audio signal; and a shotgun audiosignal.

The apparatus caused to generate the at least one output audio signalbased on the parameter associated with the move and the at least oneaudio signal may be further caused to analyse the at least one audiosignal based on the parameter and a frequency band associated with theat least one audio signal.

The apparatus caused to generate the at least one output audio signalbased on the parameter associated with the move and the at least oneaudio signal may be further caused to analyse the at least one furtheraudio signal based on the parameter and a frequency band associated withthe at least one audio signal.

The parameter may comprise a rotation of the first part relative to thesecond part.

The apparatus caused to generate the at least one output audio signalbased on the parameter associated with the move and the at least oneaudio signal may be further caused to: generate a mono audio signalbased on a rotation of the first part relative to the second part and/oran orientation of the apparatus being a first configuration; andgenerate a spatial audio signal based on a rotation of the first partrelative to the second part and/or an orientation of the apparatus beinga second configuration.

The apparatus caused to generate the at least one output audio signalbased on the parameter associated with the move, the at least one audiosignal, and the at least one further audio signal may be further causedto: generate a mono audio signal based on the at least one further audiosignal; generate a spatial audio signal based on the at least one audiosignal; and combine the mono audio signal based on the at least onefurther audio signal and the spatial audio signal based on the at leastone audio signal based on the parameter.

The one of the first part or second part being able to move relative tothe other part is configured to reveal the at least one microphone suchthat the at least one audio signal captured by the at least onemicrophone signal is a spatial audio signal.

According to a fourth aspect there is provided an apparatus comprising:a first part, the first part having at least one camera configured tocapture images; a second part having at least one microphone configuredto capture at least one audio signal, wherein one of the first part orsecond part is able to move relative to the other part; means fordetermining a parameter associated with the move; means for generatingat least one output audio signal based on the parameter associated withthe move and the at least one audio signal.

According to a fifth aspect there is provided a computer programcomprising instructions [or a computer readable medium comprisingprogram instructions] for causing an apparatus comprising: a first part,the first part having at least one camera configured to capture images;a second part having at least one microphone configured to capture atleast one audio signal, wherein one of the first part or second part isable to move relative to the other part, to perform at least thefollowing: determining a parameter associated with the move; generatingat least one output audio signal based on the parameter associated withthe move and the at least one audio signal.

According to a sixth aspect there is provided a non-transitory computerreadable medium comprising program instructions for causing an apparatuscomprising: a first part, the first part having at least one cameraconfigured to capture images; a second part having at least onemicrophone configured to capture at least one audio signal, wherein oneof the first part or second part is able to move relative to the otherpart, to perform at least the following: determining a parameterassociated with the move; generating at least one output audio signalbased on the parameter associated with the move and the at least oneaudio signal.

According to a seventh aspect there is provided an apparatus comprising:a first part, the first part having at least one camera configured tocapture images; a second part having at least one microphone configuredto capture at least one audio signal, wherein one of the first part orsecond part is able to move relative to the other part, to perform atleast the following: circuitry configured to determine a parameterassociated with the move; circuitry configured to generate at least oneoutput audio signal based on the parameter associated with the move andthe at least one audio signal.

According to an eighth aspect there is provided a computer readablemedium comprising program instructions for causing an apparatuscomprising: a first part, the first part having at least one cameraconfigured to capture images; a second part having at least onemicrophone configured to capture at least one audio signal, wherein oneof the first part or second part is able to move relative to the otherpart, to perform at least the following: determining a parameterassociated with the move; generating at least one output audio signalbased on the parameter associated with the move and the at least oneaudio signal.

An apparatus comprising means for performing the actions of the methodas described above.

An apparatus configured to perform the actions of the method asdescribed above.

A computer program comprising program instructions for causing acomputer to perform the method as described above.

A computer program product stored on a medium may cause an apparatus toperform the method as described herein.

An electronic device may comprise apparatus as described herein.

A chipset may comprise apparatus as described herein.

Embodiments of the present application aim to address problemsassociated with the state of the art.

SUMMARY OF THE FIGURES

For a better understanding of the present application, reference willnow be made by way of example to the accompanying drawings in which:

FIG. 1 shows schematically examples of multi-microphone captureapparatus capable of capturing audio signals and images with a fixedcamera;

FIG. 2 shows schematically further examples of multi-microphone captureapparatus capable of capturing audio signals and images with a rotatablecamera part in two positions according to some embodiments;

FIG. 3 shows schematically a spatial audio system according to someembodiments;

FIG. 4 shows a flow diagram of the operation of the spatial audio systemas shown in FIG. 3 according to some embodiments;

FIGS. 5 and 6 show schematically further examples of multi-microphonecapture apparatus capable of capturing audio signals and images with arotatable camera part according to some embodiments;

FIG. 7 shows schematically an example of multi-microphone captureapparatus capable of capturing audio signals and images with a rotatablecamera, wherein at least one of the microphones are coaxially mountedwith the camera;

FIG. 8 shows example microphone effective locations for examplemulti-microphone capture apparatus capable of capturing audio signalsand images with a rotatable camera capable of changing the effectivelocation;

FIG. 9 shows example microphone effective locations for a furtherexample multi-microphone capture apparatus capable of capturing audiosignals and images with a rotatable camera part capable of changing theeffective location;

FIG. 10 shows example microphone effective locations for an additionalexample multi-microphone capture apparatus capable of capturing audiosignals and images with a rotatable camera part capable of changing theeffective location;

FIG. 11 shows schematically a further spatial audio system according tosome embodiments;

FIG. 12 shows a flow diagram of the operation of the spatial audiosystem as shown in FIG. 11 according to some embodiments;

and

FIG. 13 shows schematically an example device suitable for implementingthe apparatus shown in previous figures.

EMBODIMENTS OF THE APPLICATION

The following describes in further detail suitable apparatus andpossible mechanisms for spatial audio signal capture and processingusing capture apparatus.

As discussed above a device with a camera that may rotate with respectto the rest of the device (and any microphones located on the device) isone which is not currently well configured with respect to spatial audiocapture.

As discussed spatial audio with correct directions in a plane can becaptured using 3 microphones and all directions can be captured using 4microphones. Many different capture and direction analysis algorithmsexist.

If a spatial audio signal is accurately captured then the spatial audiosignal can be rotated.

Rotating cameras enable many different use cases. Some microphonelocations are more optimal for some use cases and different microphonelocations are needed for others. Use case examples include:

Teleconferences where the capture device is located on a table and thecamera can be configured to turn to an active speaker;

Video recording where the user holds the capture device in their hand;and

Selfie recording where the camera can be configured to be turned towardsthe user.

Additionally there is further complexity in that the capture device maybe operated in either portrait or landscape orientation.

As discussed above spatial audio directions are typically determinedbased on an analysis of level and phase/time differences of microphoneaudio signals. The configuration of the physical device defines thedirections and the distances of the microphones which affects thecaptured audio signals. This can therefore sometimes help allowinglarger differences to the microphone audio signals where there is alarge physical distance between the microphones, but it can also be ahindrance. In some configurations the microphone locations are such thatthe acoustic waves which are converted into the audio signals may travelaround the device using several paths that are close similar but haveslightly difference level and phase/time differences. This produceserrors in any level and phase/time difference estimations and disturbsdirection detection.

An example of designed microphone configurations is shown in FIG. 1. Theexample capture device 101 on the left hand side of FIG. 1 shows a fixedconfiguration of a camera 115 mounted at the top left of one face of thedevice 101. The capture device furthermore comprises 3 microphones, afirst microphone 114 located at the top middle of the same face as thecamera 115, a second microphone 112 located at the bottom middle of thesame face as the camera 115 and a third microphone 116 located at thetop middle of the face opposite to the camera 115. This configurationattempts to optimized spatial capture when the device is operated in alandscape orientation because it has 3 microphones in approximately ahorizontal plane (when the device is in landscape orientation) thatoptimizes capture of directions in a horizontal plane which is importantfor spatial audio.

This configuration is also designed such that the microphones are awayfrom corners where users typically hold the device 101 when operating inlandscape orientation.

The example capture device 103 on the right hand side of FIG. 1 shows afixed configuration of a camera 125 mounted at the same top left of oneface of the device 103. The capture device however in this designcomprises 3 microphones, a first microphone 124 located at the topmid-right of the same face as the camera 115, a second microphone 112located at the top mid-left of the same face as the camera 115 and athird microphone 116 located at the top middle of the face opposite tothe camera 115. This configuration attempts to optimize spatial audiocapture in portrait orientation as it has 3 microphones in approximatelya horizontal plane (when the device is operated in portrait orientation)that optimizes capture of directions in a horizontal plane which isimportant for spatial audio. The microphones are for the same reasonsabove located away from the device bottom where users typically hold thedevice when using it in portrait orientation.

As indicated before audio direction detection is often based onanalysing differences between microphone signals. The capture deviceitself may disturb this analysis because the microphones are no longerin free-field conditions. Often optimal location for the microphones fordirection detection is to be near edges where they are closer tofree-field conditions than in the middle of a facet. In particular, ifmicrophones in a pair are further away from an edge than is theirdistance from each other then there may be problems in performingdirection analysis.

Rotating camera product concepts have been introduced by severalmanufacturers which reduces the number of cameras needed in a mobiledevice. For example, the requirement for a selfie camera would becomeredundant as well as large field-of-view cameras to some extent.

When a capture device captures both (spatial) audio signals and video orimage data, these need to be aligned or to match. Where sound sourcedirections in the audio signal do not match a corresponding visualobject direction in the image or video data, the resulting playback ofvideo and audio signals will be incorrect and may produce the effect oflooking at a subject talking form a first direction but hearing theirvoice from a second difference direction creating a perceptual errorwhich may make the user disorientated or dizzy. In a device where thecamera may rotate the captured audio signals need to be modified to keepthe alignment or match between audio and video object directions. Inaddition, mobile devices may be used in many orientations (landscape,portrait, on a table etc.) and therefore the audio alignment has to takemany differing inputs into consideration.

For example if the device has only 3 microphones that do not rotate withthe camera, the problem is that not all use cases can be servedoptimally regardless of the microphone locations.

Furthermore if the device has 4 microphones, then there may be a goodlocation configuration for microphones so that all use cases can beserved but the microphones may be positioned such that the user mayeasily cover them with their hands or the microphones may be located inplaces where detecting some directions is difficult.

In configurations with 5 microphones, then there may be configurationsor designs where the microphone locations are not easily covered by theuser's hands and most if not all directions can be detected withsuitable accuracy.

One proposal as explored in further detail within the embodimentsdescribed herein is to place all of the microphones on the rotatingcamera part. However the rotating camera part is typically small andplacing all of the microphones on this part although would enable thealignment between microphone audio signals and thus the spatial audiosignals and the images captured by the camera would solve the audiorotation issue above. However the design of locating the camera andmicrophones on the rotating part causes the microphones to be generallytoo close and therefore the audio signals from the microphones arehighly correlated. Highly correlated audio signals produce a poorerresult for several reasons but particularly beamforming at lowerfrequencies or for direction analysis at low frequencies.

The concepts which are described with respect to the followingembodiments include:

specific microphone locations for a device with a rotating camera;

analyzing audio directions differently in different camera rotations;

analyzing audio directions similarly for low frequencies independent ofcamera rotation and analyzing directions for high frequenciesdifferently depending on camera rotation; and

creating spatial and non-spatial audio depending on camera rotation anddevice orientation.

Additionally a rotating camera part with some microphones located on itcan make direction analysis difficult when microphones move with respectto each other.

Additionally microphones have sound ports to allow the acoustic waves topass to the microphones and furthermore require a hole or aperture in adevice cover to prevent a significant attenuation of the acoustic wavesby the cover. These holes in the cover are problematic as the currenttrend of making edge-to-edge displays leaves little space for holes.Additionally a user's hands can cover these holes or create handlingnoise issues as hands move over the holes. Holes are furthermore in somesituations deemed aesthetically problematic and cause designers tolocate them where they less apparent but typically acousticallyproblematic, such as next to a camera which may have noisy moving parts.

The concept as discussed herein in the embodiments is a capture devicewith microphones and rotating camera which can be configured toimplement spatial audio capture and alignment with the camera images insuch a manner that the user watching and listening is provided with aquality audio signal which is perceptually in alignment with any videoimages and additionally is able to suitably place holes or ports.

A first example capture device is shown in FIG. 2. The capture device inthis example is a mobile device with 4 microphones located within thedevice body main part and a camera located in a rotating camera partaxially rotatable from the main part. The device configuration as shownin FIG. 2 having the microphones located in the main part is thereforeconfigured not to rotate with the camera. The mobile device isconfigured to receive acoustic waves from substantially all directionsusing the microphones to generate audio signals and from thesemicrophone audio signals create a rotated spatial audio signal wheresound object directions match or are aligned with corresponding videoobject directions (from the camera) regardless of the device orientationand camera part orientation. In some embodiments audio directiondetection is implemented based on the direction or orientation of thecamera part because the camera part rotation affects microphone signaldelays and levels.

The Focusing can be done using beamforming or spatial filtering whereaudio directions are analysed, and audio is amplified/attenuated basedon analysed directions. These three microphones make it possible toanalyse audio directions in a plane but since the device can be usedboth in portrait and landscape orientations, we need directionalanalysis also in a direction perpendicular to that plane. Thus 4microphones in total are needed.

FIG. 2 therefore shows as an example of such a configuration. A mobiledevice 200 which comprises a body on which the components can be mountedor located. The body can be divided into a main part 210 and camera part220. The main part 210 and camera part 220 are able to be rotate aboutan axis (show as dashed line 240). The mobile device 200 shown on theleft hand side of the figure shows when the camera part 220 is rotated90 degrees from alignment with the main part 210 and on the right handside of the figure shows the camera part 220 aligned with the main part210 such that when in alignment the camera part and the main part appearsubstantially a single object.

In the examples shown in FIG. 2 the microphones are located on the mainpart 210. This makes manufacturing the device easier because themicrophones are in the same part where the device processor andmotherboard are and thus wirings or other connections are shorter andless complex (and do not have to negotiate the rotating axis).

In FIG. 2 is shown a first microphone, microphone 1, 218 located at thetop centre of the main part 210 on a first face of the body of themobile device. In the examples shown herein a face (of the body) of themobile device is a side with larger dimensions such as height and widthor larger area as compared to one of the edges of the body of the mobiledevice which is the side with a narrow dimension such as thickness orsmaller area.

A second microphone, microphone 2, 212 is located at the bottom centreof the main part 210 on the first face of the body of the mobile device.In other words the first and second microphones are separated by thelength (the longest dimension) of the main part of the mobile device.

A third microphone, microphone 3, 214 is located at the top centre ofthe main part 210 on a second face of the body of the mobile device,where the second face is the opposite side to the first face. The firstand the third microphones are separated by the thickness of the mobiledevice.

A fourth microphone, microphone 4, 216 is located at the top left of themain part 210 on the first face of the bode of the mobile device. Thefirst and fourth microphones are separated by half the width (theremaining dimension) of the main part of the mobile device.

Additionally is shown the rotating camera part 220 comprising a camera205 located on a face of the rotating camera part such that when therotating camera part 220 is aligned with the main part 210 the camera205 and the first, second and fourth microphone are on the same ‘face’.However in some embodiments the camera 205 can be located on the edge ofthe mobile device, or any other suitable location on the rotating camerapart.

The first microphone 218, second microphone 212 and third microphone 214are located such that they can be used to capture spatial audio when thedevice is operated in landscape mode when spatial audio horizontaldirections are enough, and the first microphone 218, second microphone212, third microphone 214 and fourth microphone 216 used if elevationdirectionality is required.

When the device is used in portrait mode the first microphone 218, thirdmicrophone 214 and fourth microphone 216 can be employed to capturespatial audio.

As the rotating camera part is close to the first microphone 218, fourthmicrophone 216 and third microphone 214 and the changing device shapemay affect audio direction detection and beamforming the device may takethe camera part orientation or direction into account and analysedirections differently and use different beams depending on the camerapart orientation.

For example as in the configuration in FIG. 2 the second microphone 212is located further away from the camera part compared to the firstmicrophone 218, fourth microphone 216 and third microphone 214 anyanalysis that uses only second microphone 212 or uses the secondmicrophone 212 in a pair with another microphone may implement ananalysis in a same way for all camera directions (in other words is notaffected or modified by the orientation of the camera part). Inaddition, as low frequencies are less disturbed by small rotating partssuch as cameras any analysis may be implemented with no modification forall camera part orientations in low frequencies even if higherfrequencies have modifications to processing based on the camera partorientations.

Direction analysis is typically done by finding the delay that givesmaximum correlation between microphones. That delay is the TimeDifference of Arrival (TDOA) and known methods (e.g. multilateration,and the methods as discussed in WO2018/060549, US20130044884) are usedto get the directions. As the microphones are not in a free-fieldenvironment and the device obstructs sound (and thus create audioshadows), the determined directions may need to be modified based on thedevice obstruction. The modification typically is a (look-up) tablewhere an input is a determined or calculated direction and the output isa final output directions. The table is usually created by testing thedevice by playing sounds from different directions, calculating theestimated directions and then comparing two actual and estimateddirections and putting the comparison result into the table. There maybe a single table for directions at low frequencies but there may bedifferent tables for directions at high frequencies each tablerepresenting different camera rotation. Also, the number of tables maydepend on the microphones used in the directional determination. Thusfor example when the microphones are located far away from each other orfar away from the rotating camera part there may be fewer tables thanfor microphones that are close to each other and/or close to therotating part.

For example with respect to FIG. 3 is shown a suitable spatial audiosignal generation system according to some embodiments.

The system shows the input microphone audio signals 300 being passed toan audio direction analyser 301.

The system may comprise an audio direction analyser 301 configured toreceive the input microphone audio signals 300 and determine audiodirections based on any suitable direction determination. The directions302 can then be passed to the audio directions modifier 303.

The system may comprise an audio directions modifier 303. The audiodirections modifier 303 may be configured to receive the directions fromthe audio direction analyser 301 and furthermore receive a camerarotation input 304 which provides a rotation parameter of the camerapart relative to the main part of the body of the device. The audiodirections modifier 303 as discussed above may use a series of look-uptables or any other suitable method to modify directions (for variousfrequency bands) based on the rotation parameter. Modification of theaudio directions can be implemented in any suitable manner. For exampleusing a parametric audio system such as the methods as discussed inWO2018/060549, US20130044884, DirAC (Directional Audio Coding) the audiodirections may simply be rotated. With ambisonics the ambisonics signalmay be rotated using ambisonics rotation matrices

The modified direction values 306 may then be passed to the spatialaudio generator 305.

The system may comprise a spatial audio generator 305. The spatial audiogenerator 305 is configured to receive the modified directions 306 andfurthermore the input microphone audio signals 300. The spatial audiogenerator 305 is configured to generate suitable (transport) audiosignals and furthermore metadata comprising the modified directions 306and pass the audio stream 308 to the multiplexer 307.

The system may comprise a multiplexer 307. The multiplexer 307 isconfigured to receive the video input 310 from the camera and the audiostream 308 and multiplex these to form the output stream 312.

A flow diagram showing the operations of the system shown in FIG. 3 isfurther shown in FIG. 4.

Thus an operation performed by the system is obtaining audio signalsfrom a microphone array as shown in FIG. 4 by step 401.

A further operation is one of obtaining camera part rotation parametersas shown in FIG. 4 by step 403.

Additionally a further operation is obtaining video signals from acamera as shown in FIG. 4 by step 400.

Having received the audio signals then a further operation is analyzingaudio signals to determine directions as shown in FIG. 4 by step 405.

Having determined the directions and obtained the camera partorientation then the directions can be modified based on the camera partorientation parameters as shown in FIG. 4 by step 407.

Then the spatial audio signals can be generated based on the modifieddirections and the audio signals from the microphones as shown in FIG. 4by step 409.

Having generated the spatial audio signals these can then be multiplexedwith the video signals as shown in FIG. 4 by step 411.

Then the multiplexed signals can be output, to be stored and/ortransmitted as shown in FIG. 4 by step 413.

In some embodiments the mobile device can be equipped with 3 microphonesand a rotating camera part. As in the embodiments described above themicrophones do not rotate with the camera part.

Thus if the number of desired audio use cases is reduced, then thedevice may have less than 4 microphones.

In FIG. 5 there are shown two further configurations are shown. Thus forexample on the left hand side is shown a mobile device 501 where thecamera 505 is located in the camera part 515.

Furthermore the example apparatus 501 comprises a first microphone,microphone 1, 516 located at the top left of the main part 512 on afirst face of the body of the mobile device. A second microphone,microphone 2, 518 is located at the top right of the main part 512 onthe first face of the body of the mobile device. In other words thefirst and second microphones are separated by the width of the main partof the mobile device.

A third microphone, microphone 3, 514 is located at the top centre ofthe main part 512 on a second face of the body of the mobile device,where the second face is the opposite side to the first face.

This configuration may be optimized for device operation in portraitmode. This configuration can focus to all camera directions and createspatial audio in portrait orientation but since there is no left-rightseparation between the microphones in landscape the device 501 can onlyproduce non-spatial audio for landscape videos even if the audio isfocused using beamforming etc

On the right hand side of FIG. 5 is shown a second mobile device 503where the camera 505 is located in the camera part 515 and themicrophones are located in the main part 512.

The device specifically comprises a first microphone, microphone 1, 528located at the top centre of the main part 512 on a first face of thebody of the mobile device. A second microphone, microphone 2, 522 islocated at the bottom centre of the main part 512 on the first face ofthe body of the mobile device. In other words the first and secondmicrophones are separated by the length of the main part 512 of themobile device.

A third microphone, microphone 3, 524 is located at the top centre ofthe main part 512 on a second face of the body of the mobile device,where the second face is the opposite side to the first face. The firstand the third microphones are therefore separated by the thickness ofthe device.

The device shown in this configuration may therefore be effective forlandscape operations. This device 503 can produce spatial audio inlandscape but can only create spatial audio signals in some cameradirections in portrait. The device 503 can focus to any direction on ahorizontal plane (in landscape) but this is of little avail because thecamera rotates on a vertical plane. Therefore, for only two camerarotation directions can the device 503 create audio focused to thecamera direction.

These devices 501 and 503 can be employed for many uses. For device 501non-spatial audio (typically mono audio) can be adequate for landscapevideos if the device main use is for teleconferencing and for device 503being able to focus to only two camera directions may be enough whenthose directions are those of a typical phone main camera and selfiecamera directions.

Spatial audio can be created using known methods. For example, 3microphones can be used to create spatial audio (binaural, stereo, 5.1etc) using methods in WO2018/060549. This type of parametric spatialaudio can be rotated by modifying the direction parameter alpha. Also,known methods can be used to create Ambisonics audio. Ambisonics audiocan be rotated by multiplying the multichannel (typically 4 channels infirst order Ambisonics) signal with rotation matrices.

The first device 501 orientation (portrait or landscape) devicemicrophones capture spatial audio that is rotated to match camera viewdirection and rotated spatial audio and camera images can be combined tocreate a video and the second device 503 orientation (landscape orportrait) microphones capture non-spatial audio and non-spatial audioand camera images are combined to create a video. In some embodimentsaudio direction detection is done differently in different cameradirections because camera part rotation affects microphone signal delaysand levels.

In some embodiments the mobile device can be equipped with more than 3microphones and a rotating camera part. As in the embodiments describedabove the microphones do not rotate with the camera part. For example,FIG. 6 shows an example device which has 5 microphones. This mobiledevice 601 comprises 5 microphones and a rotating camera. The mobiledevice 601 is configured to detect audio directions from substantiallyall directions using microphone signals and creates rotated spatialaudio signal where sound object directions in audio match correspondingvideo objects directions in video regardless of the device orientationand camera direction.

The microphone locations are such that there are at least 4 microphonesin areas that users don't typically touch and microphones are not farfrom device edges in both landscape and portrait orientations. Thus forexample the device specifically comprises a first microphone, microphone1, 624 located at the top right of the main part 612 on a first face ofthe body of the mobile device. A second microphone, microphone 2, 618 islocated at the bottom centre of the main part 612 on the first face ofthe body of the mobile device. A third microphone, microphone 3, 626 islocated at the top right of the main part 612 on a second face of thebody of the mobile device, where the second face is the opposite side tothe first face. The first and the third microphones are thereforeseparated by the thickness of the device. A fourth microphone,microphone 4, 614 is located at the bottom centre of the main part 612on a second face of the body of the mobile device. The second and thefourth microphones are therefore separated by the thickness of thedevice. A fifth microphone, microphone 5, 616 is located at the topcentre of the main part 612 on the first face of the body of the mobiledevice. The first and the fifth microphones are therefore separated byhalf the width of the device.

In some embodiments audio direction detection is implemented differentlyin different camera directions because camera part rotation affectsmicrophone signal delays and levels.

In the example device as shown in FIG. 6 microphones 1, 3, and 5 areapproximately on the same plane as the camera rotation and are thus goodfor focusing audio to any direction where the camera can be rotated.Microphones 2, 4, and 5 are approximately on a horizontal plane when thedevice is in landscape orientation and are therefore good for capturingspatial audio in landscape orientation. In this configuration it is onlymicrophone 5 which is not near a device edge when camera is in defaultposition but this is less of a problem because microphone 5 is mostimportant for capturing spatial audio when the device is employed in alandscape orientation and the other microphones for this (microphones 2,4) are located relatively far from microphone 5 and therefore a pairwisecomparison of microphone signals produces good directional sensitivity.Furthermore this configuration enables elevation directions to bedetermined when the device is in an landscape orientation employingmicrophone 1 in addition to microphones 2, 4, and 5. Although thelocation of microphone 5 is not near a device edge and thus theelevation sensitivity (when compared pairwise to microphone 1) is not asgood as a configuration where the microphone is located at the edge (andhas a greater separation) this is not as significant as human beings areless sensitive to accuracy of directions in elevation.

In some embodiments at least two of the microphones are located on therotatable camera part. For example in some embodiments the mobile devicehas at least 3 microphones that are positioned so that they aresignificantly in a plane with respect to the camera regardless of camerapart rotation. The mobile device thus is able to capture or recordspatial audio with microphones from both parts in one orientation andfocused audio with microphones in the camera part in anotherorientation.

For example as shown in FIG. 7, in some embodiments, three microphones(or more) are placed into a device with a rotating camera so that themicrophones form a plane with the camera. In such embodiments two of themicrophones rotate with the camera so that the plane always staysaligned with the camera direction. The three microphone signals are thenused to create spatial audio using a suitable method.

For example the device, as shown in FIG. 7 comprises a first microphone,microphone 1, 716 located at the end of the rotating camera part 715 ofthe body of the mobile device. A second microphone, microphone 2, 718 islocated at the opposite end of the rotating camera part 715 of the bodyof the mobile device. A third microphone, microphone 3, 714 is locatedat the bottom centre of the main part 712 on a first face of the body ofthe mobile device. The first and the second microphones are thereforeseparated by the width of the device. The first and the third and thesecond and the third are separated within one dimension by approximatelythe length of the device and in a perpendicular dimension byapproximately half the width of the device.

The generation of the spatial audio signals from the audio signals canbe performed for example based on the methods as described inUS20130044884. The analysis for directions described in US20130044884can remain fixed in all camera rotations because the plane ofmicrophones rotates with the camera.

In some embodiments, audio can be focused to a direction that isdependent on the camera directions. Typically, if the camera is zoomedthen audio would be focused towards camera view direction. Focusing canbe done using a suitable beamforming operation. In such an operationmicrophones from the camera part are only used for beamforming becausethe beamforming direction is dependent on the microphone configurationand microphones located in the rotating camera part stay in the sameconfiguration with respect to the camera.

In some embodiments where the camera is located or housed in therotating camera part of the mobile device the camera part issignificantly smaller than the rest of the device. The mobile devicecomprises at least 3 microphones, at least two microphones located incamera part and at least one camera located in the main part. In theseembodiments the mobile device can be configured to use at least onemicrophone from the camera part and at least one microphone from therest part to analyse audio directions for low frequencies and at leasttwo microphones from the camera part to analyse audio directions forhigh frequencies. The analysed directions and microphone signals canthen be used to create spatial audio signals. In some embodiments themobile device may use the rotating camera part microphones in order tofocus audio to at least one direction where the direction is dependenton the camera part rotation.

In some embodiments there may be configurations other than shown in FIG.7, but there are always at least 2 microphones located in the camerapart and at least 1 microphone in the main part. When audio directionsare analysed based on phase or level differences between microphonesignals the performance is better for the lower frequencies when usingthe bigger distance between microphones, for example the distancebetween one camera part microphone and one main part microphone when themain part microphone is located at the bottom of the mobile devicebecause then the level and phase differences are bigger. Also, at lowerfrequencies the camera part is relatively smaller when compared to thewavelength of audio than at high frequencies and furthermore even whenthe camera part is rotated this rotation has less effect on the analysisas much as for the higher frequency analysis.

The analysis can in some embodiments be rotation independent in someembodiments. In some embodiments the analysis may differ for some camerarotations but only a small number or few different analyses are neededto obtain the values needed for rotation compensation.

For higher frequencies the rotating camera part is likely to disturb theanalysis when done in different rotations when both camera partmicrophones and main part microphones are used because the wavelength ofthe audio is closer to the dimensions of the camera part. Therefore, forhigher frequencies a directional analysis should be implemented usingonly camera part microphone audio signals. In such embodiments there maybe an additional benefit in that since the camera part rotates both thecamera and the microphones the analysis can be the same for all camerarotations when we are interested in audio directions with respect to thecamera as is usually the case.

The analysed directions and microphone signals can then be used tocreate a spatial audio signal as in US20130044884.

In some embodiments the mobile device is configured to analyse audiodirections for some camera rotations using microphones from both partsand for other camera rotations using microphones only from the camerapart. These analysed directions together with microphone signals areused to create spatial audio.

As the rotating camera part changes the way sound travels around thedevice when the camera part is rotated it would be expected that audiodirection analysis is dependent on the camera rotation. As indicatedthis may require a large number of different analysis for manydirections. In some embodiments the analysis is thus performeddifferently for only a number of fixed camera rotations, for example 0,90, 180, and 270 degrees that are most commonly used for example forselfies, landscape videos etc. Microphones from both the rotating camerapart and the main part are used for analysing audio directions in thesefixed camera rotations for best accuracy in all frequencies. The sameanalysis may be used around these fixed directions so that for examplethe analysis that is used for 0 degrees is also used for rotationsaround 0 degrees i.e. from −20 to 20 degrees rotations. Outside thefixed rotations and their neighbourhood, the device switches to usingonly rotating camera part microphones for direction analysis. This isbecause those microphones rotate with the camera and a single analysiscan be used for all camera directions.

In some embodiments two or more microphones in the camera part are usedfor audio focusing (typically beamforming) and spatial audio is createdusing both the main and rotating part microphones or only the main partmicrophones. Spatial audio and focused audio are combined to create anaudio signal that is spatial but that emphasizes the focused direction.

In some embodiments the audio can be focused to a direction using anysuitable method. For example in some embodiments the focussing isimplemented by beamforming. Beamforming (and most if not all of theother known focussing methods) requires that microphone locations areknown and the locations with respect of the camera are known if thefocus direction is desired to be relative to the camera direction.Therefore in some embodiments beamforming can be implemented usingcamera part microphones only since they rotate with the camera and theirlocations are thus fixed relative to the camera and each other.

Spatial audio typically requires some separation between microphones sothat the microphone signals contain the natural decorrelation of theacoustic space where the recording is done. This is, in particular, truefor a mobile device that uses an omnidirectional microphone.(Omnidirectional microphones are used because they are cheaper and moreresilient to wind noise than directional microphones.)

In some embodiments a mobile device comprising a rotating camera partand main part may be configured such that an effective microphonelocation changes as the camera rotates because the sound port of themicrophone is revealed under the rotating camera part. This enables thesame microphone to be used in different use cases to create spatialaudio where use cases require different microphone locations.

As microphones are typically located inside the mobile device, there isa sound port which is configured to connect the microphone to theoutside the device so that sounds from outside the device can reach themicrophone. For spatial audio capture an important characteristic of themicrophone is how the microphone reacts differently for sounds fromdifferent directions. The sound port and the sound port hole (theinterface between the sound port and the exterior of the mobile device)in the cover of the device is important because outside the port soundsfrom different directions travel different paths but inside the soundport all sounds travel the same path. Therefore, sound port locationsdefine the microphone effective location for spatial audio capture.

An example of this may be shown in FIG. 8. FIG. 8 for example shows anexample mobile device 801 with a rotating camera part 807 and a mainpart 803. The rotating camera part 807 comprises a camera 805 located atthe end of the rotating camera part 807. Additionally is shown twomicrophones located within the main part. A first microphone, mic2, 815is located in the main part with a sound port 807 which exits the mobiledevice main part on one of the faces. This is representative of a normalmicrophone with a sound port which has the same effective location forall camera part rotations. Thus the effective location of the microphoneis constant and independent of the rotation of the rotating camera part807.

Additionally is shown a further microphone, mic 1, 817. The furthermicrophone is coupled to a sound port which is shown as a port 827 whichhas a first exit at a first location 821 at the face of the main bodywhen the rotating camera part is closed and a second location 819 overthe microphone on the top surface of the main part 803. This secondlocation 819 is blocked when the rotating camera part is in closedposition (or in an aligned angle) with respect to the main part (inother words when both parts are aligned with each other) and is openwhen the rotating camera part is in an open or non-aligned angle (whenone part is not aligned with the other). This causes the effectivelocation of the microphone to change as the second location 819 isexposed.

Thus, for example, when the rotating camera part is closed themicrophones 1 and 2 effective locations 821 and 807 are respectively tothe left and right from the camera axis and the microphones can be usedto capture stereo audio. The rotating camera part may be closed in twopossible camera directions. If the camera is pointing “right” (top-rightin the figure) then mic 1 817 is to the right from the axis and mic 2815 is to the left and they came be used to capture left and right audiosignals. When the camera is pointing “left” (bottom-left in the figure)then mic 1 817 is to the left and mic 2 815 is to the right of thecamera axis and they can be used to capture left and right audiosignals.

When the camera part is open as shown in FIG. 8, then the effectivelocation 819 of mic 1 817 is changed and now mic 1 817 is the leftmicrophone and mic 2 815 is the right microphone. Both microphones areto the right of the camera axis but this small sideways shift is notsignificant for stereo audio capture as long as there is a left/rightseparation with respect to the camera axis between the microphones.Again, if the camera points 180 degrees to the opposite directioncompared to FIG. 8, then mic 2 815 becomes the left microphone and mic 1817 becomes the right microphone.

When there are at least 3 microphones, spatial audio can be createdusing known methods such as presented in WO2018/060549, US20130044884.The changing effective microphone locations is taken into account bychanging the direction calculation according to the current effectivemicrophone locations.

In some embodiments the effective microphone location changes when thecamera rotates because the sound port of the microphone is revealedunder the rotating camera part. The effective location changes graduallyas the camera part moves. This enables the same microphone to be used indifferent use cases to create spatial audio where use cases requiredifferent microphone locations.

These embodiments are similar to those described above but in theseembodiments the microphone effective location changes gradually and canbe configured to be defined for a range of camera rotations.

For example FIG. 9 shows a mobile device 901 with a rotating camera part907 and a main part 903. The rotating camera part 907 comprises a camera905 located at the end of the rotating camera part 807. Additionally isshown a microphone located within the main part, mic1, 917 with a soundport 921 which is formed as an open slot or groove within the main part.The open slot or groove has one end at the microphone 917 and anotherend at one of the faces of the main part. This open slot or groove issuch that as the rotating camera part rotates from a closed or alignedangle with respect to the main part (when both parts are aligned witheach other) to an open or non-aligned angle (when one part is notaligned with the other) then the effective location of the microphonechanges as a different part of the groove is exposed. Thus the effectivelocation of the microphone is a first location at the face of the mainbody when the rotating camera part closed and a second location over themicrophone and at the opposite end of the open slot when the rotatingcamera part is open. The slot may be any suitable shape, for examplecurved or straight.

The example as shown in FIG. 9 shows a single microphone and its groove,but typically devices would have at least two microphones and theirgrooves.

For example as shown in FIG. 10, a plan view of an example mobile deviceis shown where the rotating camera part 1005, shown by the dashedoutline, is at various angles with respect to the main part 1003, shownby the solid outline. In this example there is shown on the rotatingcamera part 1005 the camera 1999. Additionally in the main part 1003 islocated a first microphone 1006 with a first groove 1008 which extendsto a first face of the mobile device. Furthermore is shown a secondmicrophone 1002 with a second groove 1009 which extends to a second faceof the mobile device.

As shown in FIG. 10, top left 1001, when the mobile device rotatingcamera part 1005 is closed the camera has a first orientation, the firstmicrophone 1006 has an effective location 1007 at the first face of themobile device, and the second microphone 1002 with an effective location1003 at the second face of the mobile device as the grooves arecompletely covered. In this configuration the microphones are thus ableto capture left audio using the second microphone 1002 and right audiousing the first microphone 1006.

Opening the rotating camera part 1005, as shown in FIG. 10, top middle1011, the camera has a second orientation, the first microphone 1006 hasan effective location 1017, and the second microphone 1002 with aneffective location 1013 as the grooves are partially exposed. In thisconfiguration the microphones are thus able to capture left audio usingthe second microphone 1002 effective location 1013 and right audio usingthe first microphone 1006 effective location 1017.

Rotating further the rotating camera part 1005, as shown in FIG. 10, topright 1021, the camera has a third orientation, the first microphone1006 has an effective location 1027, and the second microphone 1002 withan effective location 1023 as the grooves are further exposed. In thisconfiguration the microphones are thus able to capture left audio usingthe second microphone 1002 effective location 1023 and right audio usingthe first microphone 1006 effective location 1027.

At a further rotation the rotating camera part 1005, as shown in FIG.10, bottom left 1031, has a fourth orientation, the first microphone1006 has an effective location 1037 which shows that almost the lengthof the groove is exposed, and the second microphone 1002 with aneffective location 1033 which is almost the length of its grooveexposed. In this configuration the microphones are thus able to captureleft audio using the second microphone 1002 effective location 1033 andright audio using the first microphone 1006 effective location 1037.

Finally in FIG. 10 is shown when the rotating camera part is rotated tobe perpendicular to the main part as shown by the bottom right 1041. Thecamera has a fifth orientation, the first microphone 1006 has aneffective location at 1027 which is the location of the microphone, andthe second microphone 1002 with an effective location 1043 which is thelocation of the microphone as the grooves are fully exposed back to thelocation of the microphones. In this configuration the microphones arethus able to capture left audio using the second microphone 1002effective location 1023 and right audio using the first microphone 1006effective location 1027.

In other words as shown in FIG. 10 the microphones can always be used tocapture stereo audio that fits the camera view because the microphoneeffective locations are always to the left and right of the camera axis.

Furthermore in some embodiments instead of changing the effectivelocations of microphones the rotating camera part may reveal (whenopened) microphones that were hidden under the camera part. Thesemicrophones may then be used in use cases where the rotating camera partis open.

More than two microphones may be used, and more than one microphoneeffective location may be changed when camera part rotates. Differenteffective locations may make possible different audio capture. Forexample, 3 microphones may be used to capture spatial audio as describedin WO2018/060549.

In some embodiments the switched position and the groove positionembodiments may be combined so that some microphones are revealed whencamera rotates and other microphone effective locations are changed.Some microphones may therefore have new effective locations and revealedmicrophones may be used together to create spatial audio or focus audioto a direction using beamforming.

In some embodiments the mobile device may furthermore be able to controlthe processing of the audio signals based on the camera rotation basedon the above sound port examples.

Thus for example there may comprise a microphone signal input 1100 andcamera rotation input 1104 which is received by camera rotation effect(on microphones) determiner 1101. The camera rotation effect determiner1101 may be configured to determine which microphones are available andnot covered by the camera and pass these audio signals 1102 to the audioalgorithm modifier (for current effective location and microphones)1103.

The audio algorithm modifier 1103 having received the processed audiosignals 1102 can then modify the algorithm to calculate directions sothat they match the camera direction and output a processed audio signalto the spatial audio generator 1105.

The spatial audio generator 1105 is then configured to generate spatialaudio signals 1108 based on the audio signals from the microphones andbased on the directions and pass these to a multiplexer 1107. In someembodiments the spatial audio generator 1105 is configured to selectmicrophone signals (depending on camera direction) to use with thedirections.

The multiplexer 1107 may be configured to receive the spatial audiosignals 1108 and the video input 1110 and multiplex the two to generatethe output data stream 1112.

The operation of the system shown in FIG. 11 is shown in the flowdiagram in FIG. 12.

Thus an operation performed by the system is obtaining audio signalsfrom a microphone array as shown in FIG. 12 by step 1201.

A further operation is one of obtaining camera part rotation parametersas shown in FIG. 12 by step 1203.

Additionally a further operation is obtaining video signals from acamera as shown in FIG. 12 by step 1200.

Having received the audio signals then a further operation is that ofdetermining for current camera rotation parameters which microphones arerevealed and/or affected (in other words covered or partially covered)as shown in FIG. 12 by step 1205.

Having determined which microphones are revealed and/or covered then theaudio algorithm is modified for the current effective location andmicrophones as shown in FIG. 12 by step 1207.

Then the spatial audio signals can be generated based on the modifiedalgorithms and the audio signals from the microphones as shown in FIG.12 by step 1209.

Having generated the spatial audio signals these can then be multiplexedwith the video signals as shown in FIG. 12 by step 1211.

Then the multiplexed signals can be output, to be stored and/ortransmitted as shown in FIG. 12 by step 1213.

In the examples above the apparatus comprising the first part, the partwith a camera, and the second part, the part with at least onemicrophone are configured to rotate or move relative to a common axis(which may for example be equipped to pass signals between the parts).In some embodiments the motion of the first part relative to the secondpart, or vice versa is not about a common axis. For example the motioncan be any translation and/or rotation. For example the camera part mayrise or periscope from the body part and then furthermore be able torotate or further move relative to the body part. This motion can beconsidered to be a motion and/or rotation of at least one of the twoparts relative to a reference point.

With respect to FIG. 13 an example electronic device which may be usedas any of the apparatus parts of the system as described above. Thedevice may be any suitable electronics device or apparatus. For examplein some embodiments the device 1700 is a mobile device, user equipment,tablet computer, computer, audio playback apparatus, etc. The device mayfor example be configured to implement the encoder/analyser part 101 orthe decoder/synthesizer part 105 as shown in FIG. 1 or any functionalblock as described above.

In some embodiments the device 1700 comprises at least one processor orcentral processing unit 1707. The processor 1707 can be configured toexecute various program codes such as the methods such as describedherein.

In some embodiments the device 1700 comprises a memory 1711. In someembodiments the at least one processor 1707 is coupled to the memory1711. The memory 1711 can be any suitable storage means. In someembodiments the memory 1711 comprises a program code section for storingprogram codes implementable upon the processor 1707. Furthermore in someembodiments the memory 1711 can further comprise a stored data sectionfor storing data, for example data that has been processed or to beprocessed in accordance with the embodiments as described herein. Theimplemented program code stored within the program code section and thedata stored within the stored data section can be retrieved by theprocessor 1707 whenever needed via the memory-processor coupling.

In some embodiments the device 1700 comprises a user interface 1705. Theuser interface 1705 can be coupled in some embodiments to the processor1707. In some embodiments the processor 1707 can control the operationof the user interface 1705 and receive inputs from the user interface1705. In some embodiments the user interface 1705 can enable a user toinput commands to the device 1700, for example via a keypad. In someembodiments the user interface 1705 can enable the user to obtaininformation from the device 1700. For example the user interface 1705may comprise a display configured to display information from the device1700 to the user. The user interface 1705 can in some embodimentscomprise a touch screen or touch interface capable of both enablinginformation to be entered to the device 1700 and further displayinginformation to the user of the device 1700. In some embodiments the userinterface 1705 may be the user interface for communicating.

In some embodiments the device 1700 comprises an input/output port 1709.The input/output port 1709 in some embodiments comprises a transceiver.The transceiver in such embodiments can be coupled to the processor 1707and configured to enable a communication with other apparatus orelectronic devices, for example via a wireless communications network.The transceiver or any suitable transceiver or transmitter and/orreceiver means can in some embodiments be configured to communicate withother electronic devices or apparatus via a wire or wired coupling.

The transceiver can communicate with further apparatus by any suitableknown communications protocol. For example in some embodiments thetransceiver can use a suitable universal mobile telecommunicationssystem (UMTS) protocol, a wireless local area network (WLAN) protocolsuch as for example IEEE 802.X, a suitable short-range radio frequencycommunication protocol such as Bluetooth, or infrared data communicationpathway (IRDA).

The transceiver input/output port 1709 may be configured to receive thesignals.

In some embodiments the device 1700 may be employed as at least part ofthe synthesis device. The input/output port 1709 may be coupled toheadphones (which may be a headtracked or a non-tracked headphones) orsimilar.

In general, the various embodiments of the invention may be implementedin hardware or special purpose circuits, software, logic or anycombination thereof. For example, some aspects may be implemented inhardware, while other aspects may be implemented in firmware or softwarewhich may be executed by a controller, microprocessor or other computingdevice, although the invention is not limited thereto. While variousaspects of the invention may be illustrated and described as blockdiagrams, flow charts, or using some other pictorial representation, itis well understood that these blocks, apparatus, systems, techniques ormethods described herein may be implemented in, as non-limitingexamples, hardware, software, firmware, special purpose circuits orlogic, general purpose hardware or controller or other computingdevices, or some combination thereof.

The embodiments of this invention may be implemented by computersoftware executable by a data processor of the mobile device, such as inthe processor entity, or by hardware, or by a combination of softwareand hardware. Further in this regard it should be noted that any blocksof the logic flow as in the Figures may represent program steps, orinterconnected logic circuits, blocks and functions, or a combination ofprogram steps and logic circuits, blocks and functions. The software maybe stored on such physical media as memory chips, or memory blocksimplemented within the processor, magnetic media such as hard disk orfloppy disks, and optical media such as for example DVD and the datavariants thereof, CD.

The memory may be of any type suitable to the local technicalenvironment and may be implemented using any suitable data storagetechnology, such as semiconductor-based memory devices, magnetic memorydevices and systems, optical memory devices and systems, fixed memoryand removable memory. The data processors may be of any type suitable tothe local technical environment, and may include one or more ofgeneral-purpose computers, special purpose computers, microprocessors,digital signal processors (DSPs), application specific integratedcircuits (ASIC), gate level circuits and processors based on multi-coreprocessor architecture, as non-limiting examples.

Embodiments of the inventions may be practiced in various componentssuch as integrated circuit modules. The design of integrated circuits isby and large a highly automated process. Complex and powerful softwaretools are available for converting a logic level design into asemiconductor circuit design ready to be etched and formed on asemiconductor substrate.

Programs, such as those provided by Synopsys, Inc. of Mountain View,Calif. and Cadence Design, of San Jose, Calif. automatically routeconductors and locate components on a semiconductor chip using wellestablished rules of design as well as libraries of pre-stored designmodules. Once the design for a semiconductor circuit has been completed,the resultant design, in a standardized electronic format (e.g., Opus,GDSII, or the like) may be transmitted to a semiconductor fabricationfacility or “fab” for fabrication.

The foregoing description has provided by way of exemplary andnon-limiting examples a full and informative description of theexemplary embodiment of this invention. However, various modificationsand adaptations may become apparent to those skilled in the relevantarts in view of the foregoing description, when read in conjunction withthe accompanying drawings and the appended claims. However, all such andsimilar modifications of the teachings of this invention will still fallwithin the scope of this invention as defined in the appended claims.

1. An apparatus comprising: a first part, the first part having at leastone camera configured to capture images; a second part having at leastone microphone configured to capture at least one audio signal, whereinone of the first part or second part is able to move relative to theother part; at least one processor and at least one non-transitorymemory including a computer program code, the at least one memory andthe computer program code configured to, with the at least oneprocessor, cause the apparatus at least to: determine a parameterassociated with the move; and generate at least one output audio signalbased on the parameter associated with the move and the at least oneaudio signal.
 2. The apparatus according to claim 1, wherein the firstpart or the second part is able to move relative to common referencepoint.
 3. The apparatus according to claim 1, wherein the move is atleast one of: a rotation about an axis in common between the first partand the second part; a pitch and/or yaw and/or roll between the firstpart and the second part; a movement of the first part relative to thesecond part; or a movement of the second part relative to the firstpart.
 4. The apparatus according to claim 1, wherein the apparatus isfurther configured to: multiplex the at least one output audio signaland the images captured with the camera; and output the multiplexed atleast one output audio signal and the images captured with the camera.5. The apparatus as claimed in claim 1, wherein the first part furtherhaving at least one further microphone configured to capture at leastone further audio signal, wherein the apparatus is configured togenerate at least one output audio signal based on the parameterassociated with the move and the at least one audio signal is configuredto generate the at least one output audio signal further based on the atleast one further audio signal.
 6. The apparatus as claimed in claim 5,wherein the apparatus is configured to generate the at least one outputaudio signal further based on the at least one further audio signal isconfigured to align the at least one output audio signal and the atleast one further audio signal based on the parameter associated withthe move.
 7. The apparatus as claimed in claim 1, wherein the at leastone microphone comprises at least three microphones arranged withrespect to the second part, and the apparatus is configured to generatethe at least one output audio signal based on the parameter associatedwith the move and the at least one audio signal is configured to: obtaina parameter defining the arrangement of the at least three microphones;obtain a parameter defining an orientation of the apparatus; andgenerate the at least one output audio signal further based on theparameter defining the arrangement of the at least three microphones andthe parameter defining an orientation of the apparatus.
 8. The apparatusas claimed in claim 7, wherein the apparatus is configured to generatethe at least one output audio signal further based on the parameterdefining the arrangement of the at least three microphones and theparameter defining an orientation of the apparatus is configured togenerate the at least one output audio signal for at least one frequencyband based on the parameter defining the arrangement of the at leastthree microphones and the parameter defining an orientation of theapparatus.
 9. The apparatus as claimed in claim 1, wherein the apparatusis configured to generate the at least one output audio signal based onthe parameter associated with the move and the at least one audio signalis configured to align the at least one output audio signal based on theparameter associated with the move such that the at least one outputaudio signal is aligned with the camera.
 10. The apparatus as claimed inclaim 1, wherein the apparatus is configured to generate the at leastone output audio signal based on the parameter associated with the moveand the at least one audio signal is further configured to analyse theat least one audio signal based on the parameter and a frequency bandassociated with the at least one audio signal.
 11. The apparatus asclaimed in claim 1, wherein the apparatus is configured to generate theat least one output audio signal based on the parameter associated withthe move and the at least one audio signal is further configured toanalyse the at least one further audio signal based on the parameter anda frequency band associated with the at least one audio signal.
 12. Theapparatus as claimed in claim 1, wherein the parameter comprises arotation of the first part relative to the second part.
 13. Theapparatus as claimed in claim 1, wherein the at least one output audiosignals comprises at least one of: at least one spatial audio signal; atleast one non-spatial audio signal; a mono audio signal; a beamformedaudio signal; or a shotgun audio signal.
 14. A method comprising:providing an apparatus, the apparatus comprising a first part, the firstpart having at least one camera configured to capture images; a secondpart having at least one microphone configured to capture at least oneaudio signal, wherein one of the first part or second part is able tomove relative to the other part; determining a parameter associated withthe move; and generating at least one output audio signal based on theparameter associated with the move and the at least one audio signal.15. The method as claimed in claim 14, wherein the first part furtherhas at least one further microphone configured to capture at least onefurther audio signal, wherein generating at least one output audiosignal based on the parameter associated with the move and the at leastone audio signal comprises generating the at least one output audiosignal further based on the at least one further audio signal.
 16. Themethod as claimed in claim 15, generating the at least one output audiosignal is further based on the at least one further audio signalcomprises aligning the at least one output audio signal and the at leastone further audio signal based on the parameter associated with themove.
 17. The method as claimed in claim 14, wherein the at least onemicrophone comprises at least three microphones arranged with respect tothe second part, and generating the at least one output audio signalbased on the parameter associated with the move and the at least oneaudio signal comprises: obtaining a parameter defining the arrangementof the at least three microphones; obtaining a parameter defining anorientation of the apparatus; and generating the at least one outputaudio signal further based on the parameter defining the arrangement ofthe at least three microphones and the parameter defining an orientationof the apparatus.
 18. The method as claimed in claim 17, whereingenerating the at least one output audio signal is further based on theparameter defining the arrangement of the at least three microphones andthe parameter defining an orientation of the apparatus comprisesgenerating the at least one output audio signal for at least onefrequency band based on the parameter defining the arrangement of the atleast three microphones and the parameter defining the orientation ofthe apparatus.
 19. The method as claimed in claim 14, wherein generatingthe at least one output audio signal based on the parameter associatedwith the move and the at least one audio signal comprises aligning theat least one output audio signal based on the parameter associated withthe move such that the at least one output audio signal is aligned withthe camera.
 20. The method as claimed in claim 14, wherein generatingthe at least one output audio signal based on the parameter associatedwith the move and the at least one audio signal further comprisesanalysing the at least one audio signal based on the parameter and afrequency band associated with the at least one audio signal.